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Abstract. - A variation of low density parity check (LDPC) error correcting codes defined 
over Galois fields (GF(q)) is investigated using statistical physics. A code of this type is 
characterised by a sparse random parity check matrix composed of C nonzero elements per 
column. We examine the dependence of the code performance on the value of q, for finite 
and infinite C values, both in terms of the thermodynamical transition point and the practical 
decoding phase characterised by the existence of a unique (ferromagnetic) solution. We find 
different ^-dependencies in the cases of C = 2 and C > 3; the analytical solutions are in 
agreement with simulation results, providing a quantitative measure to the improvement in 
performance obtained using non-binary alphabets. 



Error correction mechanisms are essential for ensuring reliable data transmission through 
noisy media. They play an important role in a wide range of applications from magnetic hard 
disks to deep space exploration, and are expected to become even more important due to the 
rapid development in mobile phones and satellite-based communication. 

The error-correcting ability comes at the expense of information redundancy. Shannon 
showed in his seminal work JlO| that error-free communication is theoretically possible if 
the code rate, representing the fraction of informative bits in the transmitted codeword, is 
below the channel capacity. In the case of unbiased messages transmitted through a Binary 
Symmetric Channel (BSC), which we focus on here and which is characterized by a bit flip 
rate p, the code rate R = N/M which allows for an error- free transmission satisfies 

R<l-H 2 (p), (1) 
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Table I. - Sum (left) and product (right) in GF{4). 



e 





1 


2 


3 


® 





1 


2 


3 








1 


2 


3 

















1 


1 





3 


2 


1 





1 


2 


3 


2 


2 


3 





1 


2 





2 


3 


1 


3 


3 


2 


1 





3 





3 


1 


2 



where H 2 (p) = — p\og 2 p— (1 — p) log 2 (l —p), when both lengths of the original message N and 
the codeword M become infinite. The right hand side of (Q) is often termed Shannon's limit. 

Unfortunately, Shannon's derivation is non-constructive and the quest for practical codes 
which saturate this limit has been one of the central topics in information theory ever since. 
The current most successful code in use is arguably the Turbo code [l], although the best 
performance to date, in terms of proximity to Shannon's bound for a particular rate, has been 
achieved by variations of the low-density parity check (LDPC) code, proposed by Gallager j|. 

A variation of Gallager's code has been recently discovered independently by MacKay and 
Neal [Q; an irregular construction of this code, using a non-binary alphabet, provides the 
best error correction performance to date Q. This discovery, based on improving the code 
construction and the alphabet used by trial and error, instigated the current work, aimed at 
clarifying the role played by the alphabet used in obtaining this outstanding code performance. 
To separate the effect of code irregularity from that of the alphabet used we focus here on the 
dependence of regular constructions on the chosen alphabet. To some extent this complements 



our previous investigation on the impact of code irregularity on the system's performance 1 13 
in the case of binary alphabets. 

Using a non-binary alphabet based on Galois fields GF(q) is carried out in the following 
manner: The sender first converts the Boolean message vector £ B of dimensionality N where 
£, B G (0, 1), Vi, to an N/b dimensional vector of GF{q = 2 b ) elements; where each segment 
of b consecutive bits is mapped onto a GF(q) number p"). The GF(q) vector is then encoded 
to an M/b dimensional GF(q) codeword Zq, in the manner described below, which is then 
reconverted to an M dimensional Boolean codeword z B , transmitted via a noisy channel. 
Corruption during transmission can be modelled by the noise vector £ B , where corrupted 
bits are marked by the value 1 and all other bits are zero, such that the received corrupted 
codeword takes the form z B = z B + £ B (mod 2). The received corrupted Boolean message is 
then converted back to a GF(q) vector z, and decoded in the GF(q) representation; finally 
the message estimate is interpreted as a Boolean vector. 

Firstly, we briefly explain the mapping of binary vectors onto the Galois field GF(q) 
elements. These represent a closed set of q elements which can be added and multiplied utilizing 
an irreducible polynomial composed of Boolean coefficients. For instance, the irreducible 
polynomial for GF(A) is x 2 + x + 1. Then, identifying the b(= 2) components of the binary 
vector with Boolean coefficients of a b—1 degree polynomial, 3©1 = (x+l) + l (mod 2) = x = 2 
and 3 (g> 2 = (x + 1) x x (mod 2) = -1 (mod 2) = 1, setting x 2 + x + 1 = (mod 2). Table 
I summarises the sum and product operations in Galois field GF{A). Secondly, we explain 



( 1 ) Binary vectors will be denoted by a superscript B; other vectors are in the GF(q) representation. 
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the encoding/decoding mechanism using regular LDPC code in the GF(q) representation. 
This is based on a randomly constructed sparse parity check matrix A of dimensionality 
(M — N)/b x M/b. This matrix is characterised by C and K nonzero GF(q) elements per 
column/row. The choice of C, K is linked to the code rate R, obeying the relation C/K = 1—R. 

Nonzero elements in each row are independently and randomly selected from a specific 
distribution that maximises the entropy of the vector AC, (all operations of vectors in the 
GF(q) representation will be carried out as defined for this field; for brevity we do not introduce 
different symbols to denote these operations) when £ is the GF(q) representation of the binary 
random noise vector C, B . Then, one constructs an M/b x N/b generator matrix G T , which is 
typically dense, satisfying AG T = ||. 

Using this matrix, encoding is carried out in the GF{q) representation by taking the product 
Zq = G T £; encoding is performed by taking the product of the parity check matrix A and the 
received corrupted message z — Zq + £, which yields the syndrome vector J = Az = AC,. The 
most probable estimate of the noise vector n is defined using the equation 

An = J, (2) 

via the iterative method of Belief Propagation (BP) Qj. This has been linked, in the case of 
Boolean codes, to the TAP (Thouless, Anderson, Palmer) based solution of a similar physical 
system ||, a relation which holds also in the case of GF(q) codes as will be shown elsewhere. 

The noise vector estimate is then employed to remove the noise from the received codeword 
and retrieve the original message £ by solving the equation G T £ = z — n. 

The similarity between error-correcting codes and physical systems was first pointed out by 
Sourlas in his seminal work [JOJ, by considering a simple Boolean code, and by mapping the 
code onto well studied Ising spin systems. We recently extended his work, which focused on 
extensively connected systems, to the case of finite connectivity ||. Here, we generalise these 
connections to spin systems in which the interaction is determined using the GF(q) algebra. 

In order to facilitate the current investigation, we first map the problem to that of a l GF(q) 
spin system' of finite connectivity. The syndrome vector J is generated by taking sums of the 

relevant noise vector elements J M = A^Ch +. . .+ A^i K C,i K , where C = (Ci=i M/b) represents 

the true channel noise; the indices ii,...,iic correspond to the nonzero elements in /i-th row 
of the parity check matrix A = (A u k). It should be noted that the noise components d 
are derived from a certain distribution P pr (Ci)i representing the nature of the communication 
channel; this will serve as our prior belief to the nature of the corruption process. This implies 
that the most probable solution of Eq. (0) corresponds to the ground state of the Hamiltonian 

M/b 

W(n)= V {ii,i2,...,i K ) {^-5[J{t 1 ,t 2 ,...,r K )\A^ 1 n ll +. . . + A l , iK n lK \) - - ^ In P pr (ni), 

(u,»a,...,ij<-) i=1 

(3) 

in the zero temperature limit (3 = 1/T — > oo. Elements of the sparse tensor "Du u i 3t , ..,i K ) take 
the value 1 if all the corresponding indices of parity matrix A are nonzero in some row - [i, 
and otherwise. The last expression on the right relates to the prior probability of the noise 
vector elements. Note that operations between vectors/elements in the GF{q) representation 
(e.g., within the 5 function) are carried out as defined in this field. 

The delta function provides 1 if the contribution for the selected site A^n^ +. . ■+A /J ,i K rii K 
is in agreement with the corresponding syndrome value J(i lt i 2> ,,, t i K \, recording an error, and 
otherwise. Notice that this term is not frustrated as there are M/b degrees of freedom while 
only (M — N)/b constraints arise from Eq. ([|), and its contribution can therefore vanish at 
sufficiently low temperatures. The choice of (3 — > oo imposes the restriction (El), limiting the 
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solutions to those for which the first term of (||) vanishes, while the second term, representing 
the prior information about the noise, survives. 

The optimal estimator, minimising the expectation of discrepancy per noise bit, is of the 
form n% — argmax agG j^ 1J ^ (S(rii, o))fl_> 00 - This is known as the marginal posterior maximiser 
(MPM) pi and corresponds to the finite temperature decoding at Nishimori's temperature 
studied in other codes |l2|, [| |). Notice that here, due to the hard constraints imposed on the 
dynamical variables, decoding at zero temperature is optimal, as the true posterior distribution 
(given J) relates to the ground state of Hamiltonian (|J), similar to other LDPC codes |(|. The 

macroscopic quantity m = (b/M) (j2il/i A serves as the performance measure. 



To eliminate the dependency of the syndrome Jn lt ... t i K ) on the noise vector £ we employ the 
gauge transformation rij — > rij + Q, J(i 1 .... i i K ) — > 0. Rewriting Eq. (J3j) in this gauge moves the 
dependency on £ to the second term where it appears in a decoupled form ( 1 / f3) In P pr (rij + Q) . 
The remaining difficulty comes from the complicated site dependency caused by nontrivial 
GF(q) algebra in the first term. However, one can rewrite this dependency in a simpler form 

S[Q;A liil m 1 +-.. + A liiK n iK ] = ^ S [0; A x a x + ■ ■ ■ + A K a K ] (4) 

Ai,...,Aif,ai,...,ajc-0 

x5{A ll A llll ) . . .8{A K ,A^ iK ) x 5(oi, n»J . . . S(a K ,m K ) , 

by introducing Kroncker's 5 and the dummy variables A\, . . . , Ak and ai, . . . , ax- 

Since codes of this type are usually used for long messages with N = 10 3 — f 5 , it is natural 
to resort to the methods of statistical mechanics for analysing their properties. The random 
selection of sparse tensor D, identifying the nonzero elements of A, and the noise vector 
£ introduces quenched disorder to the system. More specifically, we calculate the partition 
function Z (V, A, £) = Tr n exp [—/3H] averaged over the disorder and the statistical properties 
of the noise estimation, using the replica method @, ||. Taking [3 — > oo gives rise to a set of 
order parameters 

b M/b I " \ 

Q ai ,a 2 ,...,a n = Jj ( Zi II ( S ( a <x> n i*))p^co ) > ( 5 ) 



M 

i=l 



where a = 1, . . . , n represents the replica index and a a runs from to q — 1 , and the variables 
Zi come from enforcing the restriction of C connections per index i 
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To proceed further, one has to make an assumption about the symmetry of order parameters. 
The assumption made here is that of replica symmetry reflected in the representation of the 
order parameters and of the related conjugate variables: 

/n 
dPir(P ,...,P q -i)l[P aa , 
a=l 

~ n 
Qa u a 2 ,...,a n = a^J dPTT{P ,...,P q -x)J{P aa , (7) 



a=l 



where ag and ag are normalisation coefficients; 7r(_P) and tt(P) represent probability distri- 
butions for q dimensional vectors P = (Pq, . . . , Pq-i) and P = (Pq, . . . , Pq-i), respectively. 
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Unspecified integrals are performed over the region P + . . . + Pq-x = 1, P a =o,...,q~i > or 
Po + . . . + Pq-i = 1) Pa=o,...,q— l > 0- Extremising the averaged expression with respect to the 
probability distributions, one obtains the following free energy per spin 



6 

M 



C , / /g-l C 



(1 



n2) BAC = -Ext J ff[dPn(p)(ln[Y,t[PaP P r(a + 

{tt.tt} /=1 \ \ a=n/=1 

if / / g-l K 



C 



J 1=1 \ \a 1 ,...,a K =0 1 = 1 

C JdPdPir(P) 7r(P)ln^P a P Q ^| , 



(8) 



where {-) A and (•)* denote averages over the distribution of nonzero units per row in con- 
structing the matrix A and over P pr (C)j respectively. 

One can calculate the free energy via the saddle point method. Solving the equations 
obtained by varying Eq.(||) is generally difficult. However, it can be shown analytically that a 
successful solution 

q-l q-1 

tt(P) = S(P - 1) JJ S(P a ), tt(P) = S(P - 1) JJ S(P a ), (9) 

a— 1 a— 1 

which implies perfect decoding m = 1, extremises the free energy for C > 2. For C — > oo, an 
unsuccessful solution, which provides m < 1, is also obtained analytically 



7r{P) = (\[8(P a -P pr (a + 0)) , ^(P) = lis(p a -^). (10) 



\a=0 




Inserting these solutions into (||) it is found that the solution (^) becomes thermodynamically 
dominant with respect to ( |l0| ) for R < l—H2(p) independently of q; which implies that the code 
saturates Shannon's limit for C — * oo as is reported in the information theory literature Q . 

Finding additional solutions analytically is difficult, we therefore resorted to numerical 
methods. Approximating the distributions tt(P) and tt (P) by 5 x 10 3 — 3 x 10 4 sample vectors 
of P and P we obtained solutions by updating the saddle point equations (100 — 500 iterations) 
for codes of connectivity C = 2, . . . , 6 and GF(q) representation q = 2, 4, 8 and for both BSC 
and Gaussian channels. Less then 50 iteration were typically sufficient for the solutions to 
converge. Due to lack of space we present here results only for the case of the BSC; results for 
the case of Gaussian channels are qualitatively similar and will be presented elsewhere. 

Since the suggested properties are different for C > 3 and C = 2, we describe the results 
separately for the two cases. For C > 3, it turns out that Eq.(||) is always locally stable. 
However, an unsuccessful solution, approaching ( |l0|) as C — > oo, becomes thermodynamically 
dominant for sufficiently large flip rate p. As the noise level is reduced, the solution (^|) becomes 
thermodynamically dominant at a certain flip rate p = p t , and remains dominant until p — > 0. 
This implies that perfect decoding m = 1 is feasible for p < p t . However, the locally stable 
unsuccessful solution remains as well above a certain noise level p s (< pt)- 

As C — > oo, the transition point pt converges from below to Shannon's limit p c = i?^~ 1 (l — R) 
irrespective of the value of q. For finite C, pt monotonically increases with q but does not 
saturate p c . This implies that error correcting ability of the codes when optimally decoded is 
monotonically improved as q increases. 
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Fig. 1. - Extremised free energies (JSJ) obtained for q = 2, 4, 8 as functions of the flip rate p (a) for 
connectivity C = 4 (> 3) and (b) for C = 2 codes with the same code rate R = 1 — C/if = 1/2. 
In both codes, broken lines represent free energies of successful solution (H), while markers stand for 
unsuccessful solutions (m < 1). Monte Carlo methods with 5 x 10 3 — 3 x 10 samplings at each step are 
employed for obtaining the latter with statistical fluctuations smaller than the symbol size. For each q 
value, the solution having the lower free energy becomes thermodynamically dominant. In (a), crossing 
points provide the critical flip rate for pt being 0.106, 0.108 and 0.109 (within the numerical precision) 
for q = 2,4 and 8, respectively, monotonically approaching Shannon's limit p c = H~ 1 (l/2) = 0.109. 
The inner figure focuses on the vicinity of the spinodal points p s , determining the limit of successful 
practical decoding. This shows p 3 to decrease with increasing q. (b) shows that C = 2 codes exhibit 
continuous transitions between the successful and unsuccessful solutions. The critical flip rate pb, 
pointed by arrows, is increases with q, while it is still far from Shannon's limit. 



The behaviour of the spinodal point p s is quite different, as shown in Fig. la, presenting 
the dependence of p t and p s on q for connectivity C = 4 (> 3). It appears that p s is 
generally decreasing with respect to q (except for unique pathological cases), indicating a 
lower practical corruption limit for which BP/TAP decoding will still be effective. Above this 
limit BP/TAP dynamics is likely to converge to the unsuccessful solution due to its dominant 
basin of attraction ||. In contrast, C — 2 codes exhibit a different behaviour; the solution 
becomes the unique minimum of free energy (^) for sufficiently small noise levels, which 
implies that practical decoding dynamics always converges to the perfect solution. However, as 
the noise level increases, the solution loses its stability and continuously bifurcates to a stable 
suboptimal solution. Unlike the case of C > 3, this bifurcation point pb, which monotonically 
increases with q, determines the limit of practical BP/TAP decoding. The practical limit 
obtained is considerably lower than both Shannon's limit and the thermodynamic transition 
point p t for other C > 3 codes with the same q value (Fig. lb). Therefore, the optimal 
decoding performance of C = 2 codes is the worst within this family of codes. 

However, p^ can become closer to, and even higher than, the spinodal point p s of other 
C > 3 codes for large q values, (Table II) implying that the practical decoding performance 
of C — 2 codes is not necessarily inferior to that of C > 3 codes. This is presumably due to 
the decreasing solution numbers to Eq.(||) for C — 2 as q increase, compared to the moderate 
logarithmic increase in the information content, tipping the balance in favour of the perfect 
solution. This may shed light on the role played by C — 2 elements in irregular constructions. 
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Table II. - The critical noise level, below which BP/TAP-based decoding works successfully, for 
different connectivity values C in the case of q = 8 and R = 1 — C/K = 0.5. This is determined 
as the spinodal point p a and the bifurcation point pt for C > 3 and C — 1, respectively. The critical 
noise for C — 2 becomes higher than that of C > 5. 
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In summary, we have investigated the properties of LDPC codes defined over GF{q) within 
the framework of statistical mechanics. Employing the replica method, one can evaluate 
the typical performance of codes in the limit of infinite message length. It has been shown 
analytically that codes of this type saturate Shannon's limit as C — > oo irrespective of the 
value of q, in agreement with results reported in the information theory literature ||. For 
finite C, numerical calculations suggest that these codes exhibits two different behaviours for 
C > 3 and C = 2. For C > 3, we show that the error correcting ability of these codes, when 
optimally decoded, is monotonically improving as q increases; while the practical decoding 
limit, determined by the emergence of a suboptimal solution, deteriorates. On the other hand, 
C = 2 codes exhibit a continuous transition from optimal to sub-optimal solutions at a certain 
noise level, below which practical decoding dynamics based on BP/TAP methods converges 
to the (unique) optimal solution. This critical noise level monotonically increases with q and 
becomes even higher than that of some codes of connectivity C > 3, while the optimal decoding 
performance is inferior to that of C > 3 codes with the same q value. This may elucidate the 
role played by C — 2 components in irregular constructions. 

Future directions include extending the analysis to irregular Gallager codes as well as to 
regular and irregular MN code |Q, [| in the Galois representation. 

We acknowledge support from the RFTF program of the JSPS (YK), EPSRC (GR/N00562) 
and The Royal Society (DS). 
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